fix(PLU-373): make notion database test row-order insensitive by awalker4 · Pull Request #714 · Unstructured-IO/unstructured-ingest

awalker4 · 2026-05-21T18:27:54Z

Summary

Notion's database query endpoint doesn't guarantee stable row order, but test_notion_source_database compared the downloaded HTML byte-for-byte. Every prior "fix" (fix(fixtures): update SharePoint and Notion integration test fixtures #693, Fix databricks sdk version #605) was just reshuffling the same four rows.
Add unordered_table_html_equality_check that compares the header positionally and data rows as a multiset of their text content, and wire it into the database test only (the page test isn't a table).
On mismatch the new check prints the symmetric difference so the next real regression won't be a black box like https://github.com/Unstructured-IO/unstructured-ingest/actions/runs/26238030726/job/77216703553 was.
Version bumped to 1.6.2-dev to satisfy check-version without claiming a release slot for a test-only change.

Test plan

uncategorized_connectors_int_test Notion tests pass (real verification — needs the CI's NOTION_API_KEY)
Smoke-tested locally against the real fixture: reordered rows compare equal, a tampered row fails with a readable diff

Summary by cubic

Make the Notion database integration test row-order insensitive and correct a misleading helper comment, eliminating flaky failures when Notion returns rows in different orders. Addresses PLU-373.

Bug Fixes
- Added unordered_table_html_equality_check to compare the header positionally and treat table rows as a multiset; prints a readable symmetric diff on mismatch.
- Used in test_notion_source_database only; the page test remains unchanged.
- Fixed the helper comment to state it scans all <tr> elements, not just the first table.
- Bumped version to 1.6.6-dev and updated CHANGELOG.md; test-only change with no runtime impact.

^{Written for commit 87d92cc. Summary will update on new commits. Review in cubic}

The notion_database integration test compared the downloaded HTML byte- for-byte against a checked-in fixture, but Notion's database query endpoint doesn't guarantee a stable row order — every prior fixture refresh was just reshuffling the same rows. Compare rows as a multiset of text content and print the symmetric difference on mismatch so real regressions aren't silent. Version is bumped to 1.6.2-dev to satisfy check-version without claiming a release slot for a test-only change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cubic-dev-ai

1 issue found across 4 files

_{Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic}

Comment said the helper inspected the first <table>, but the code scans every <tr> in the document via soup.find_all("tr"). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 1 file (changes from recent commits).

_{Shadow auto-approve: would auto-approve. This change only modifies test infrastructure and documentation: it adds an order-insensitive HTML table equality check for a single Notion integration test, updates CHANGELOG, and bumps a dev version, with no impact on production code, business logic, or critical systems.

Re-trigger cubic}

potter-potter · 2026-05-26T15:45:04Z

@claude please review

claude · 2026-05-26T15:45:23Z

Claude finished @potter-potter's task —— View job

Code Review

Reviewing PR changes to fix flaky Notion database tests...

Todo List:

Read and analyze all changed files
Review the new unordered_table_html_equality_check function
Assess test integration and approach
Check version bump and changelog
Provide comprehensive review feedback

Analysis of Changes

✅ New `unordered_table_html_equality_check` Function (`equality.py:51-91`)

Strengths:

Addresses the root issue: Notion's API doesn't guarantee stable row ordering, and this function correctly handles that by treating data rows as a multiset
Header preservation: Smart decision to compare the first <tr> positionally as a header while treating subsequent rows as unordered
Excellent error reporting: The symmetric difference output (lines 85-89) will make debugging much easier than previous black-box failures
Clean implementation: Uses Counter for multiset operations and sorted() for consistent comparison
Proper scope: Helper function split_rows() is appropriately scoped and reusable

✅ Test Integration (`test_notion.py:65`)

Correctly applied only to test_notion_source_database where row order is unstable
Appropriately left test_notion_source_page unchanged since pages aren't tables
Clean import and usage

✅ Version and Documentation

Version bump (__version__.py): Appropriate bump to 1.6.2-dev for test-only change
Changelog (CHANGELOG.md:1-5): Well-documented entry clearly stating this is test-only with no published behavior impact

Code Quality Assessment

Security: ✅ No security concerns - test infrastructure only
Performance: ✅ Efficient use of Counter and sorted() for multiset comparison
Maintainability: ✅ Clear function name, good docstring, readable implementation
Testing: ✅ Addresses flaky test issue with deterministic comparison

Minor Observations

Error message formatting: The current format is clear and actionable:

only in expected (x2): row content here  
only in current  (x1): different row content

Edge case handling: Function gracefully handles empty tables (if not rows: return "", [])
Text extraction: Uses get_text(" ", strip=True) for consistent whitespace handling

Recommendation

✅ LGTM - This is a well-implemented fix that directly addresses the flaky test issue. The approach is sound, the implementation is clean, and the error reporting improvement is valuable for future debugging.

The fix correctly identifies that Notion's database query endpoint doesn't guarantee stable row order and implements an appropriate order-insensitive comparison while preserving header semantics.

potter-potter

LG

…row-order # Conflicts: # CHANGELOG.md # unstructured_ingest/__version__.py

awalker4 requested a review from a team as a code owner May 21, 2026 18:27

awalker4 temporarily deployed to ci May 21, 2026 18:28 — with GitHub Actions Inactive

awalker4 mentioned this pull request May 21, 2026

feat(PLU-161): flatten_metadata option for databricks_volume_delta_tables #712

Merged

3 tasks

cubic-dev-ai Bot reviewed May 21, 2026

View reviewed changes

Comment thread test/integration/connectors/utils/validation/equality.py Outdated

plu-373 fix misleading comment on unordered table check

6899ef1

Comment said the helper inspected the first <table>, but the code scans every <tr> in the document via soup.find_all("tr"). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

awalker4 temporarily deployed to ci May 22, 2026 15:29 — with GitHub Actions Inactive

cubic-dev-ai Bot reviewed May 22, 2026

View reviewed changes

potter-potter approved these changes May 26, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into austin/notion-stable-…

87d92cc

…row-order # Conflicts: # CHANGELOG.md # unstructured_ingest/__version__.py

awalker4 temporarily deployed to ci May 26, 2026 16:02 — with GitHub Actions Inactive

awalker4 temporarily deployed to ci May 26, 2026 16:03 — with GitHub Actions Inactive

awalker4 merged commit 6689db0 into main May 26, 2026
37 of 38 checks passed

awalker4 deleted the austin/notion-stable-row-order branch May 26, 2026 16:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(PLU-373): make notion database test row-order insensitive#714

fix(PLU-373): make notion database test row-order insensitive#714
awalker4 merged 3 commits into
mainfrom
austin/notion-stable-row-order

awalker4 commented May 21, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

potter-potter commented May 26, 2026

Uh oh!

claude Bot commented May 26, 2026 •

edited

Loading

Uh oh!

potter-potter left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

awalker4 commented May 21, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Summary by cubic

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

potter-potter commented May 26, 2026

Uh oh!

claude Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review

Analysis of Changes

✅ New unordered_table_html_equality_check Function (equality.py:51-91)

✅ Test Integration (test_notion.py:65)

✅ Version and Documentation

Code Quality Assessment

Minor Observations

Recommendation

Uh oh!

potter-potter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

awalker4 commented May 21, 2026 •

edited by cubic-dev-ai Bot

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

claude Bot commented May 26, 2026 •

edited

Loading

✅ New `unordered_table_html_equality_check` Function (`equality.py:51-91`)

✅ Test Integration (`test_notion.py:65`)